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DETAILED ACTION 
Continued Examination Under 37 CFR 1.114 

1. A request for continued examination under 37 CFR 1.114, including the fee set 
forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this 
application is eligible for continued examination under 37 CFR 1.114, and the fee set 
forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action 
has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 
6/19/2006 has been entered. 

Claim Rejections - 35 USC § 102 

2. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 

3. Claims 1, 2, and 5 are rejected under 35 U.S.C. 102(b) as being anticipated by 
State et al. (NPL Document "Superior Augmented Reality Registration by Integrating 
Landmark Tracking and Magnetic Tracking", herein referred to as "State"). 

As per claim 1 , State teaches the claimed: 

1 (Currently amended) A three-dimensional image display method 
comprising: 

detecting a position of a light source existing in real space; 
By teaching of: 

The (real) light source is tracked (by the mechanical 
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arm) 

(pg. 437, 3 rd paragraph under section 8) 

(emphasis added to various quotations from the reference) 

Here, the position of the light source is detected through the tracking of the mechanical 

arm. 

State teaches the claimed: 

comparing the position of the light source and a virtual position of a 
display object in a three-dimensional image displayed in real space to 
obtain a relative positional relation therebetween; and 

By teaching of: 

Figure 8 demonstrates a virtual object, a knot, casting a 
shadow on a real object, a sculpture. The geometry of the 
sculpture was digitized with the mechanical arm and placed in 
the scene. The (real) light source is tracked (by the mechanical 
arm), and the shadow map is calculated in real-time 
(pg. 437, 3 rd paragraph under section 8) 

A tracked light source moves real and virtual shadows in sync, 
(pg. 436, caption under figure 8) 

Here, in order for the light source to cause virtual shadowing behind the three- 
dimensional image (where the three dimensional image is the knot) the positions of both 
the light source and the object would have to be known. This is required in order to 
form the correct virtual shadows in response to the light cast from the light source and 
drawing the virtual shadowing caused by the three dimensional object (also see figure 8 
where this is shown). 
State teaches the claimed: 

shading in the three-dimensional image 
By teaching of: 
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Figure 9 shows a similar scene. The knot intersects the real 
objects, emphasizing the accurate registration of the synthetic 
imagery (the knot and its shadow) with the real cuboids, 
(pg. 437, 3 rd paragraph under section 8) 

Here, figure 9 shows shading on the three-dimensional image itself (where the three 

dimensional image is the knot). 



As per claim 2, State teaches the claimed: 

2. (Original) The method according to claim 1, further comprising: 
detecting lightness of the light source. 

By teaching of: 

CCD video cameras ... attached to the HMD 
(pg. 430, 1st paragraph under section 4) 

The second component of the landmark finder is the image 
analyzer, which starts its search for a landmark by inspecting 
the search area defined by the landmark predictor, 
(pg. 432, 1 st paragraph under section 6.2.2) 

For our specific camera and frame grabber hardware, and under 
the lighting conditions in our lab, such a simple algorithm can 
reliably distinguish between only a small number of different 
colors 

(pg. 432, 2nd paragraph under section 6.2.2) 
Here, by inspecting a search area using an image analyzer, the lightness is detected as 
indicated in the reference where different colors are detected. Since some colors 
appear darker or lighter than others the lightness is detected in this analysis. 



As per claim 5, this claim is similar in scope to claim 1 , and is rejected under the same 
rationale. 
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Claim Rejections - 35 USC § 103 

4. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

5. Claims 6-10 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
State. 

As per claim 6, the reasons and rationale for the rejection of claim 1 is incorporated 
herein. 

State does not explicitly teach the claimed: 

a plurality of detectors which detects a position of a light source existing in 
real space; 

However, State suggests the claimed limitation by teaching of in figure 6 of two 
detectors (cameras labeled d and c2) used to calculate a position of a landmark. It 
would have been obvious to one of ordinary skill in the art to incorporated this position 
detection method using multiple detectors to detect a position of a light source as well. 
The modification can be achieved by replacing the landmark labeled (L2) in figure 6 with 
a light source, and using detectors d and c2 to detect the light source position. One 
advantage to such a modification is to allow for a more freely controlled light source, 
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which is not constrained by a mechanical arm for position detection, by utilizing a 
plurality of detectors to detect the light source position. 

As per claims 7 and 8, State teaches the claimed: 

a display surface configured to display the three-dimensional image 

By teaching of: 

Head-Mounted Displays (HMDs) 

(pg. 429, 1 st paragraph under section 1) 

Here, the head-mounted display has an associated display surface or surfaces near the 

eyes that produce the three-dimensional image for the user to view. 

State does not explicitly teach the claimed: 

the detector is disposed on at least one of the display surface and a 
surface adjacent to the display surface. 

the detector is disposed to be adjacent to the display surface. 

However, States suggests the claimed limitations by teaching of: 

CCD video cameras ... attached to the HMD 
(pg. 430, 1st paragraph under section 4) 

Here, the video cameras are located adjacent to the display and are capable of 

detecting of light. It would have been obvious to one of ordinary skill in the art to 

achieve the claimed limitation by utilizing the CCD video cameras as light position 

detectors which are adjacent to the display surface. The reference suggests this use of 

the CCD video cameras for light position detection in figure 6 where cameras C1 and 

C2 are used to detect the position of landmark L2. The motivation of claim 6 is 

incorporated herein. 
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As per claim 9, State does not explicitly teach the claimed: 

9. (Currently amended) The device according to claim 5, wherein the 
detector is disposed at a position where the detector detects the 

light emitted from the light source located in the same direction as at least 
one of a display direction of the three dimensional image and a direction in 
which the three-dimensional image is observed 

However, State suggests the claimed limitation by teaching of: 

Head-Mounted Displays (HMDs) 

(pg. 429, 1 st paragraph under section 1) 

CCD video cameras ... attached to the HMD 
(pg. 430, 1st paragraph under section 4) 

where the HMD display displays the three-dimensional image in a direction in front of 

the user's eyes. It would have been obvious to one of ordinary skill in the art to achieve 

the claimed limitations by using the camera as a light position detector instead of the 

mechanical arm to achieve the claimed limitations. The modification can be achieved 

by using the two cameras (detectors) mounted on the head mounted display to detect 

the position of the light source. The result of this modification is that the light will be 

detected in the same direction as the display direction of the three-dimensional image 

since the head mounted display displays the image and because the cameras 

(detectors) are mounted on the head mounted display. The motivation of claim 6 is 

incorporated herein. 

As per claim 10, State does not explicitly teach the claimed: 

10. (Currently amended) The device according to claim 5, wherein: 
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the detector includes three-primary colors detection unit that adds colors 
to the shade. 

State suggests the claimed limitation by teaching of: 

RGB component values ... distinguish between only a small number of 
different colors 

(pg. 432, 2 nd paragraph under section 6.2.2) 
Further, State suggests of adding colors to the shade in figure 9 where a shadow is 
added over top of the yellow colored circular mark and the colors are mixed. Thus, it 
would have been obvious to one of ordinary skill in the art to achieve the claimed 
limitations by using the camera as a light position detector instead of the mechanical 
arm. The modification can be achieved by using the two cameras (detectors) mounted 
on the head mounted display to detect the position of the light source and thus also 
detect the three-primary colors. The modification is possible because the reference 
states the cameras are capable of detecting three-primary colors. The motivation of 
claim 6 is incorporated herein. 

6. Claims 3 and 4 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
State in view of Drettakis et al. (NPL Document "Interactive Common Illumination for 
Computer Augmented Reality", herein referred to as "Drettakis"). 

As per claim 3, the reasons and rationale for the rejection of claim 1 is incorporated 
herein. 

State does not explicitly teach the claimed: 
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detecting positions of a plurality of light sources existing in real space; 

Drettakis teaches the claimed limitation by teaching of: 

We have introduced a new framework for dealing with the problem of 
common illumination between real and synthetic objects and light 
sources in the context of computer augmented reality. 
(1 st paragraph under section 6.3) 

We have briefly discussed some possible future research paths, by 
removing the restrictions one by one, to achieve interactive common 
illumination for first a moving camera, then moving lights and finally 
moving real objects. 
(3rd paragraph under section 6.3) 

Here, in order to achieve interactive illumination with a CG object the positions of the 

plurality of light sources must be known in order to correctly add the realistic shadows 

effects. Without knowing the positions of the light sources the shadow effects can look 

unrealistic. Thus, the system detects a plurality of light source positions. 

It would have been obvious to one of ordinary skill in the art at the time of 

invention to combine State and Drettakis. Drettakis teaches one advantage of the 

combination by teaching of: 

Novel advances in computer vision are used for camera 
calibration and user-friendly modeling of the real scene, a recent 
interactive radiosity update algorithm is adapted to provide fast 
illumination update and finally textured polygons are used for display. 
This approach allows interactive update rates on mid-range graphics 
workstations. Our new framework will hopefully lead to CAR systems with 
interactive common illumination without restrictions on the movement 
of real or synthetic objects, lights and cameras, 
(abstract) 

where State would benefit from the added functionality. 



As per claim 4, State does not explicitly teach the claimed: 
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4. (Currently amended) The method according to claim 3, further 
comprising: 

obtaining a position of a single virtual light source, which represents the 
plurality of light sources; and 

Drettakis teaches the claimed limitation by teaching of: 

In the system presented we have not shown the addition of virtual 
lights. This is not too hard to achieve, but requires some modification to 
the incremental update approach, since the addition of a light source 
typically affects a large part of the environment. In addition, special 
attention must be taken in the re-scaling of the image before display since 
the addition of a source can add an order (or orders) of magnitude to 
the radiosity values of the scene 
(col 4, lines 10-12) 

Here, the single virtual light source can be represented as the accumulation of radiosity. 

The reference refers to "adding" light onto the total radiosity for each light source. 

When every light source is considered, one can represent this radiosity as a single 

virtual light source through the addition of all individual light sources. 

State does not teach the claimed: 

comparing the position of the virtual light source and the virtual position of 
the display object in the three-dimensional image to obtain the relative 
positional relations therebetween. 

Drettakis teaches the claimed limitation in figures 3 and 4. Figure 4 shows the result of 

the comparison between the virtual light source (as calculated as the accumulated 

radiosity in figure 3) and the virtual position of the display object (the floating box above 

the desk). The comparison of the positions and relative positional relationship result in 

the rendering of a soft and varying shadow on the desk below the floating object. The 

varying nature of this shadow can be due to the plurality of light directional components 
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provide by the virtual light source. One advantage to using this claimed feature with 
State is to achieve realistic soft shadows affects utilizing multiple light sources. 



Response to Arguments 

7. Applicant's arguments with respect to claims have been considered but are moot 
in view of the new ground(s) of rejection. More specifically, the previous prior art 
rejection of based upon the reference of Loscos has been withdrawn in response to 
amendments added by applicant that include "a three-dimensional image displayed in 
real space". However, the prior art reference of State has now been applied to address 
this limitation because the system of State uses an augmented reality system with a 
combined optical and digital display which allows the user to interact with the three- 
dimensional image in real space (bottom of 1 st col on pg. 429). 



Conclusion 

8. The following prior art made of record and relied upon is considered pertinent to 
applicant's disclosure: NPL Document "UNC Hybrid Tracking Research" 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Daniel F. Hajnik whose telephone number is (571) 272- 
7642. The examiner can normally be reached on Mon-Fri (8:30A-5:00P). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Ulka J. Chauhan can be reached on (571) 272-7782. The fax phone 
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number for the organization where this application or proceeding is assigned is 571- 
273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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